An electronic imaging system depends on an electronic image sensor to create an electronic representation of a visual image. Examples of such electronic image sensors include charge coupled device (CCD) image sensors and active pixel sensor (APS) devices (APS devices are often referred to as CMOS sensors because of the ability to fabricate them in a Complementary Metal Oxide Semiconductor process). Typically, these images sensors include a number of light sensitive pixels, often arranged in a regular pattern of rows and columns. For capturing color images, a pattern of filters is typically fabricated on the pattern of pixels, with different filter materials being used to make individual pixels sensitive to only a portion of the visible light spectrum. The color filters necessarily reduce the amount of light reaching each pixel, and thereby reduce the light sensitivity of each pixel. A need persists for improving the light sensitivity, or photographic speed, of electronic color image sensors to permit images to be captured at lower light levels or to allow images at higher light levels to be captured with shorter exposure times.
Image sensors are either linear or two-dimensional. Generally, these sensors have two different types of applications. The two-dimensional sensors are typically suitable for image capture devices such as digital cameras, cell phones and other applications. Linear sensors are often used for scanning documents. In either case, when color filters are employed the image sensors have reduced sensitivity.
A linear image sensor, the KLI-4104 manufactured by Eastman Kodak Company, includes four linear, single pixel wide arrays of pixels, with color filters applied to three of the arrays to make each array sensitive to either red, green, or blue in its entirety, and with no color filter array applied to the fourth array; furthermore, the three color arrays have larger pixels to compensate for the reduction in light sensitivity due to the color filters, and the fourth array has smaller pixels to capture a high resolution monochrome image. When an image is captured using this image sensor, the image is represented as a high resolution, high photographic sensitivity monochrome image along with three lower resolution images with roughly the same photographic sensitivity and with each of the three images corresponding to either red, green, or blue light from the image; hence, each point in the electronic image includes a monochrome value, a red value, a green value, and a blue value. However, since this is a linear image sensor, it requires relative mechanical motion between the image sensor and the image in order to scan the image across the four linear arrays of pixels. This limits the speed with which the image is scanned and precludes the use of this sensor in a handheld camera or in capturing a scene that includes moving objects.
There is also known in the art an electronic imaging system described in U.S. Pat. No. 4,823,186 by Akira Muramatsu that includes two sensors, wherein each of the sensors includes a two-dimensional array of pixels but one sensor has no color filters and the other sensor includes a pattern of color filters included with the pixels, and with an optical beam splitter to provide each image sensor with the image. Since the color sensor has a pattern of color filters applied, each pixel in the color sensor provides only a single color. When an image is captured with this system, each point in the electronic image includes a monochrome value and one color value, and the color image must have the missing colors at each pixel location interpolated from the nearby colors. Although this system improves the light sensitivity over a single conventional image sensor, the overall complexity, size, and cost of the system is greater due to the need for two sensors and a beam splitter. Furthermore, the beam splitter directs only half the light from the image to each sensor, limiting the improvement in photographic speed.
In addition to the linear image sensor mentioned above, there are known in the art image sensors with two-dimensional arrays of pixels where the pixels include pixels that do not have color filters applied to them. For example, see Sato et al in U.S. Pat. No. 4,390,895, Yamagami et al in U.S. Pat. No. 5,323,233, and Gindele et al in U.S. Pat. No. 6,476,865. In each of the cited patents, the sensitivity of the unfiltered or monochrome pixels is significantly higher than the color pixels, requiring the application of gain to the color pixels in order to match the color and monochrome signals from the pixel array. Increasing gain increases noise as well as signal, causing degradation in the overall signal to noise ratio of the resulting image. Frame in U.S. Patent Application Publication 2003/0210332 discloses a pixel array with most of the pixels unfiltered, but the color pixels suffer from the same sensitivity deficit as mentioned above.
Noda in European Patent No. 0138074 describes a video camera system using both color and panchromatic pixels. In particular, Noda discloses image processing techniques for image sensors with W, Ye, Cy and W, Ye, Cy, G color filter array patterns where W (panchromatic), Ye (yellow), Cy (cyan), and G (green) pixels are used to provide a normal red, green, and blue color image. The weighting of different types of pixels is selected to minimize moire patterns in the resultant image.
In U.S. Pat. No. 5,172,220, Beis discloses a surveillance camera that switches between gray scale (panchromatic) mode and color mode (using color pixels). In this case, the ambient illumination level is used as a global switch, simply selecting the panchromatic pixels for producing the output image when the ambient light level is low.
In U.S. Pat. No. 6,246,865, Lee discloses combination of panchromatic and color pixels to produce an image with higher dynamic range than can be captured with only the color pixels or panchromatic pixels, using the color pixels to estimate an interpolated luminance signal in portions of the image where panchromatic pixels are clipped.
For many image capture devices, the full sensor resolution exceeds the number of pixels that can be read out at normal video frame rates, and thus some form of subsampling must be used to achieve video frame rates. One way of achieving data reduction is by direct subsampling of the raw CFA data as described, for example, in U.S. Pat. No. 5,493,335 including one pattern based on 2×2 blocks of pixels. Another way of achieving data reduction is by analog combining the signals from two or more pixels before reading the CFA data from the sensor.
In U.S. Pat. No. 6,366,318, Smith discloses the generation of a regular array of CFA color values using an irregular array that was directly subsampled from a larger array of CFA color pixels.
Some prior art, such as U.S. Pat. No. 6,246,865, addresses high luminance conditions, when panchromatic pixels are clipped and the color pixels have good signal to noise ratio. Some prior art, such as U.S. Pat. No. 5,172,220, addresses very low luminance conditions. Under these conditions, the panchromatic pixels have a usable signal to noise ratio and the color pixels have an unacceptably low signal to noise ratio. There persists a need to provide an image capture system that provides improved image quality low resolution images from a two-dimensional sensor over a wide range of conditions.